In video processing systems, padding is a process of extrapolating or extending pixel values around edges of a video frame. This helps in processing of a video frame beyond its picture boundary. Padded frame is required at the time of motion estimation (ME) and motion compensation (MC). Different video processing systems adopt different data flows to create padded frame and make them available for motion estimation or motion compensation. Typically a processor performs the horizontal padding before writing out frame into external memory and does vertical padding by memory copy of top line of the frame into external memory. This method is costly in terms of memory as well as performance. In few advanced systems, hardware accelerators that require extended pixels, perform padding locally on currently processed part of the video frame. Padding operation can also be applicable for frame filling, so that received frame width becomes multiple of width of processing unit of block, for example a macro block. In such cases, padding logic units would have to be instantiated in more than one place, resulting in more area. Such a method could also impact the processing cycles of the particular hardware accelerator. There is no flexibility to experiment with data flow without performance loss inside video processor. Hence, there is a need for a common padding unit that performs padding on the fly in a video processing system with reduced memory bandwidth, with reduced processing cycles and with minimal area. It is also desirable that the padding process be generic and flexible enough to allow experimentation with data flow to create padded frame and support feature similar to padding. It should also be extendible to support padding along with other data processing such as compression and decimation.